Topics

Further thoughts on the Camera Evaluations

Geoff Boyle
 

In preparation for the next round of camera evaluations I asked for comments and got a lot regarding the type of charts I was using.

I've been thinking a lot about this and I think we may have it all backwards.

Conventional rec 709 TV has a DR of at most 6 stops and if the charts are slavishly adhered to only 5 stops.

However, once the knees and hypergammas etc are taken into account we are actually working with 8 stops at least.

10 bit log-c will contain about 11.5 stops of DR and that's more than the best available domestically HDR sets can handle.

Of course that limitation isn't the camera, it's the record format and that's why I record RAW for these tests with a few odd exceptions that are forced on me like rendering 6K to 4K before recording.

So what are the tests trying to do?

I think that we, or I, have lost focus on this.

The tests are to find out what the usable DR of the cameras is.
To establish where the highlights actually begin to clip.
To see what colour distortions appear with exposure change.
To see if there are motion artifacts created by the cameras.
To check what happens to skintone with the different cameras and different lighting.
I acknowledge that I can't check all lighting and camera combinations, although this would be a really useful experiment.

This is why I created the stress test chart, it's not designed to show how a camera reacts to rec 709 conditions, it's designed to see how far a camera can be pushed until it breaks.

THAT is what I'm interested in, it's the kind of information that will affect how I shoot and what I use.

Just as ACES is designed to work regardless of output space, that can be changed later, so the CML tests will be conducted regardless of any "Standards". We will try and establish the limitations of the cameras.
Of course some of you will only ever see those results within the limitations of a 709 or sRGB environment. They will at least show you what happens within that environment.

For those who want more I will make available both EXR's in 16 bit ACES and, eventually, the original camera files.

I am now working out how to get the most information out of these evaluations. Part of this will be using a "standard" tungsten light source, the old fashioned kind that creates light by burning something :-) It will also involve charts lit at different levels within the one shot.
Which charts is another matter!
I'm leaning towards a CML stress Test and a Macbeth on either side of the frame, one side lit "normally" and one side 3 stops over. In the middle will be a Kodak grey scale plus card as this has been my reference for a very long time. A movement test will be below the Kodak chart. If I can get a small CDM chart then I will probably squeeze it in below the Kodak chart just for the traditionalists. Oh and a CamBelles chart will feature somewhere :-)

I will also try and do some comparisons between faces lit with tungsten and LED light. This will include the DSC VFX chart.

 

 

Cheers

 

Geoff Boyle NSC FBKS

Cinematographer

Netherlands

www.gboyle.nl

 

John Brawley
 



On Jun 21, 2018, at 4:18 AM, Geoff Boyle <geoff@...> wrote:

I am now working out how to get the most information out of these evaluations. Part of this will be using a "standard" tungsten light source, the old fashioned kind that creates light by burning something :-) It will also involve charts lit at different levels within the one shot.



I always struggle with how to test the same issue.  I’ve included some grabs for a recent one I did.

I think a lot of the testing of how colour is “tracking” changes with over or under exposure can be covered through looking at the base lighting setup as you over or under expose.  So rather than having multiple charts in the same frame lit to different exposures, you can arguably compare each frame from individual exposure brackets on a normally lit chart for just looking at colour.

I don’t think you can ever look at DR on a chart though.  For that to be meaningful I want a black clip point and a white clip point.

So what I do is try to create an clipping exposure ramp.  Using a leko say slashed at a very steep angle on a white wall.  I then walk along the ramp using a meter and using some black tape, I then mark each time the stop increases +1 as I walk closer to the lamp. (Doubling of FC measure direct)

A waveform will tell you where the sensor is hard clipping now (also through a LUT or in LOG or whatever), and you always have a white point, a black point and near black.

I think this is a good way to give me the white clipping point AND as you do exposure brackets, you can see where the clip point moves as you change exposure.  You can also see what changes to grade curves happen as you change ISO on baked in codecs for example (non RAW shooting)

Then using a light trap, (little cube above the chart here below) it sets a black reference, and has a near black setting too (the black around the aperture of the light trap)

Now this doesn’t tell me how colour tracks into near clipping, but I can arguably see that by looking at the colour chart from say a plus 3 bracket and compare it to the “normal” exposure.  Being white, the same Fc exposure as far as the meter means that the white wall will already be over mid grey at the start base exposure.

Caveats in this being, it’s not super accurate, (there shouldn’t be a difference distance between +1 and +2) it’s just me roughly metering the level with the head contacting the wall and facing camera.  The angle and reflectivity may alter how far it’s spaced, but when comparing different cameras it allows me to benchmark some things.

After the first chart, which is Alexa normal (with a show LUT) assuming 800 base ISO is “normal” I included a Blackmagic Ursa Mini Pro 4.6 on the same exposure (with show LUT).  Then there’s a second RED Helium un-LUT’d un-WB frame at plus 1 (just to show the clipping difference)

Anyway, I think you get the idea of what you can do with using clipping to measure DR on a exposure ramp, because the way colour responds in near over exposure is a different (related) issue to me which I can see though using exposure brackets.

JB

John Brawley
Atlanta Georgia
DP - The Resident Season 2

Paul Curtis
 

On 21 Jun 2018, at 10:51, John Brawley <john@...> wrote:
So what I do is try to create an clipping exposure ramp. Using a leko say slashed at a very steep angle on a white wall. I then walk along the ramp using a meter and using some black tape, I then mark each time the stop increases +1 as I walk closer to the lamp. (Doubling of FC measure direct)

This is a neat idea.

In the past i've taken a density transmission wedge (stouffer do a few) and they're reasonably priced. Then build a simple box around an Area48 light (so flat emission, no LED hotspots) and shoot that for a DR. Like a poor mans DSC Xyla chart). I've always felt transmission is better than reflective and you can see the range on a single shot, or two shots for HDR cameras.

Although that's just DR and so the stress test chart being spoken about probably has more overall value.

So i'd agree with you Geoff. Testing the camera is what it's all about, to break it. In addition to your target list, also there's interest in areas like out of gamut colours and whether the camera can see, manage and handle them (brake lights at night, the kind of LED lighting you now see out and about at night) and i don't know if there's any way to include that kind of saturation/brightness in a chart?

cheers
Paul

Paul Curtis, VFX & Post | Canterbury, UK

Geoff Boyle
 

I like Johns tests a lot and are very close to what I would do if I was testing for a shoot.

 

The thing about tests in a wilderness, i.e. no specific shoot, is that they need to cover different things.

 

The reason for the secondary exposure area is to facilitate HDR tests. I should be able to hold detail in both sets of charts on a 1,000 nit monitor.

 

We will also be able to see how things change in an HDR world.

 

Still working on it 😊

 

Cheers

 

Geoff Boyle NSC FBKS

Cinematographer

Netherlands

www.gboyle.nl

 

 

Geoff Boyle
 

Good point Paul and I’ll try and find some way of including that.

 

I’m sure that there are LED toys around…

 

I will be including a cavity black as well.

 

Cheers

 

Geoff Boyle NSC FBKS

Cinematographer

Netherlands

www.gboyle.nl

 

 

also there's interest in areas like out of gamut colours and whether the camera can see, manage and handle them (brake lights at night, the kind of LED lighting you now see out and about at night) and i don't know if there's any way to include that kind of saturation/brightness in a chart?

cheers
Paul

Paul Curtis, VFX & Post | Canterbury, UK

 

Mitch Gross
 

On Jun 21, 2018, at 4:18 AM, Geoff Boyle <geoff@...> wrote:

I will also try and do some comparisons between faces lit with tungsten and LED light. This will include the DSC VFX chart.

This is the only part that raises a red flag for me. Which LED light? There are so many and their spectrums are so varied. Adding them in means that you’ll have two variables at play: the spectrum of the light and the color response of the cameras. I’m just not sure what this will prove and it seems to me like a separate but also worthy test. 

I could see you (or someone else, go ahead) conducting an equally complex series of tests based solely on this factor half a dozen popular LED lights and one true quartz tungsten unit. Half a dozen popular cameras. Subject a fair skinned person and a dark complexion person, each with different hair colors, holding or wearing various fabrics along with a color chip chart. That’s tons of data to parse all on its own, and is quite separate from the tests you’re currently planning. 

Mitch Gross
Cinema Product Manager 
Panasonic Systems Solutions Company of North America
New York


Art Adams
 

10 bit log-c will contain about 11.5 stops of DR and that's more than the best available domestically HDR sets can handle.

I'm curious where you got that number. I can see the full 14 stops in 10-bit ProRes. The bottom ones are pretty subtle, as they are in any camera, but they are there. I certainly don't think you lose 3.5 stops. 

--
Art Adams
Director of Photography
San Francisco Bay Area

Geoff Boyle
 

Simple maths Art Cineon log, which is what I meant to say, allocates 90 bits per stop. Log-c allocates 80 bits per stop so is actually 12.8.

cheers
Geoff Boyle NSC
EU based cinematographer
+31 637155076
www.gboyle.co.uk

On Thu, 21 Jun 2018, 16:32 Art Adams, <art.cml.only@...> wrote:

10 bit log-c will contain about 11.5 stops of DR and that's more than the best available domestically HDR sets can handle.

I'm curious where you got that number. I can see the full 14 stops in 10-bit ProRes. The bottom ones are pretty subtle, as they are in any camera, but they are there. I certainly don't think you lose 3.5 stops. 

--
Art Adams
Director of Photography
San Francisco Bay Area

Art Adams
 

I agree with Mitch on the LED light. At that point you're not really testing the camera, you're testing the light.

For me, a camera test only works if you use light sources that are available to everyone, because then everyone can expect the same results under those conditions. LED lights are HIGHLY variable, and not everyone has the best ones. The worst ones aren't worth testing, but there are more of those out there.

For example, Cineos are great. Are they common? No.

Also, some LED lights are better at daylight than tungsten, and some are (strangely) the other way around. Which single LED light would you choose, and at which temperature? And will everyone have access to that light? It's too many variables.

--
Art Adams
Director of Photography
San Francisco Bay Area

Art Adams
 

Well, if I shoot a Xyla chart, I see 14 stops.

As you say, f*ck the numbers, look at the pictures. :)

--
Art Adams
Director of Photography
San Francisco Bay Area

Mark Kenfield
 

It will prove how the different cameras perform under a common, trusted LED that large numbers of us use as key lights all the time (I'd vote for something along the lines of a Skypanel, Litemat or Area48 - good colour, tried-and-tested).

Somehow I suspect that selecting the lights we use, based on their spectral response relative to the specific sensors of the various camera manufacturers, is beyond all but the absolutely most technically pedantic amongst us.

We'll have a full-spectrum reference point from the tungsten source. No one's using full-spectrum daylight sources, so I think it makes sense to simply point a high-quality LED (that's common on professional sets) at the scene, and then we get a solid reference point for precisely the kind of discontinuous spectrum lighting that we're all flooding our sets with these days. We all know that they're all a little bit different, that doesn't invalidate the test.

Cheers,

Mark Kenfield
Cinematographer
Melbourne

0400 044 500

On 22 June 2018 at 00:14, Mitch Gross <mitchgrosscml@...> wrote:
I’m just not sure what this will prove and it seems to me like a separate but also worthy test. 

Geoff Boyle
 

I am intending to use the most common LED light, OK not on micro budget shoots, the ARRI SkyPanel.

If you don't think it's useful then just ignore it.

It is an issue that people have to deal with. I'd go with, actually I do go with, BB&S or Cineo, but SP's are everywhere.

I've been looking back to 2011 when i used tungsten & HMI, huge difference, and a definite win for tungsten. Not just the skin tone but the colour chart as well.

cheers
Geoff Boyle NSC
EU based cinematographer
+31 637155076
www.gboyle.co.uk


On Thu, 21 Jun 2018, 16:52 Mark Kenfield, <mark@...> wrote:
It will prove how the different cameras perform under a common, trusted LED that large numbers of us use as key lights all the time (I'd vote for something along the lines of a Skypanel, Litemat or Area48 - good colour, tried-and-tested).

Somehow I suspect that selecting the lights we use, based on their spectral response relative to the specific sensors of the various camera manufacturers, is beyond all but the absolutely most technically pedantic amongst us.

We'll have a full-spectrum reference point from the tungsten source. No one's using full-spectrum daylight sources, so I think it makes sense to simply point a high-quality LED (that's common on professional sets) at the scene, and then we get a solid reference point for precisely the kind of discontinuous spectrum lighting that we're all flooding our sets with these days. We all know that they're all a little bit different, that doesn't invalidate the test.

Cheers,

Mark Kenfield
Cinematographer
Melbourne

0400 044 500

On 22 June 2018 at 00:14, Mitch Gross <mitchgrosscml@...> wrote:
I’m just not sure what this will prove and it seems to me like a separate but also worthy test. 

alister@...
 

Simple maths Art Cineon log, which is what I meant to say, allocates 90 bits per stop. Log-c allocates 80 bits per stop so is actually 12.8.

Except below middle grey the number of bits required reduces by 50% for each stop you go down, so the lowest/darkest stops use use far fewer bits, the darkest using a hand full of bits.  Thus it is possible to record in excess of 15 stops with using the log-c curve and 10 bit. 

Alister Chapman

DoP - Stereographer
UK Mobile +44 7711 152226
US Mobile +1(216)298-1977


www.xdcam-user.com    1.5 million hits, 100,000 visits from over 45,000 unique visitors every month!  Film and Video production techniques, reviews and news.

















On 21 Jun 2018, at 15:36, Geoff Boyle <geoff@...> wrote:

Simple maths Art Cineon log, which is what I meant to say, allocates 90 bits per stop. Log-c allocates 80 bits per stop so is actually 12.8.

cheers 
Geoff Boyle NSC
EU based cinematographer
+31 637155076
www.gboyle.co.uk

On Thu, 21 Jun 2018, 16:32 Art Adams, <art.cml.only@...> wrote:

10 bit log-c will contain about 11.5 stops of DR and that's more than the best available domestically HDR sets can handle.

I'm curious where you got that number. I can see the full 14 stops in 10-bit ProRes. The bottom ones are pretty subtle, as they are in any camera, but they are there. I certainly don't think you lose 3.5 stops. 

-- 
Art Adams
Director of Photography
San Francisco Bay Area



Nick Shaw
 

This is the EI800 LogC curve. You can see it only has equal code values per stop at the top end. It rolls off significantly at the bottom. So you can see that it can code significantly more than 12.5 stops.



Nick Shaw
Workflow Consultant
Antler Post
UK

On 21 Jun 2018, at 16:01, alister@... wrote:

Except below middle grey the number of bits required reduces by 50% for each stop you go down, so the lowest/darkest stops use use far fewer bits, the darkest using a hand full of bits. Thus it is possible to record in excess of 15 stops with using the log-c curve and 10 bit.

John Brawley
 

Hi.

I personally agree that the SkyPanel is a good de-facto daylight source most will know.

WB issues / channel aside,  Tungsten tells us what we need to know for full spectral response.

John Brawley
Atlanta Georgia
DP - The Resident Season 2


On Jun 21, 2018, at 11:00 AM, Geoff Boyle <geoff@...> wrote:

I am intending to use the most common LED light, OK not on micro budget shoots, the ARRI SkyPanel.

Geoff Boyle
 

It’s looking pretty straight until about -4 stops.

 

I’ve attached a Kodak Cineon chart, looks pretty straight to me.

 

If you look at everything I’ve written about log I’ve always said that manufacturers mess with it but this is REAL log.

 

Cheers

 

Geoff Boyle NSC FBKS

Cinematographer

Netherlands

www.gboyle.nl

 

From: cml-raw-log-hdr@... <cml-raw-log-hdr@...> On Behalf Of Nick Shaw
Sent: 21 June 2018 17:36
To: cml-raw-log-hdr@...
Subject: Re: [cml-raw-log-hdr] Further thoughts on the Camera Evaluations

 

This is the EI800 LogC curve. You can see it only has equal code values per stop at the top end. It rolls off significantly at the bottom. So you can see that it can code significantly more than 12.5 stops.



Nick Shaw
Workflow Consultant
Antler Post
UK

On 21 Jun 2018, at 16:01, alister@... wrote:

Except below middle grey the number of bits required reduces by 50% for each stop you go down, so the lowest/darkest stops use use far fewer bits, the darkest using a hand full of bits. Thus it is possible to record in excess of 15 stops with using the log-c curve and 10 bit.

Geoff Boyle
 

Which is why tungsten is the main test.

 

Unless that is someone wants us to sponsor us to go to Coober Pedy SA, Australia where we’ll probably get some good clear daylight.

 

Cheers

 

Geoff Boyle NSC FBKS

Cinematographer

Netherlands

www.gboyle.nl

 

From: cml-raw-log-hdr@... <cml-raw-log-hdr@...> On Behalf Of John Brawley

.

WB issues / channel aside,  Tungsten tells us what we need to know for full spectral response.

 

 

Colin Elves
 

Unless you can get a 1KW Hive wasp that’s basically continuous daylight. 

I’m not sure who is the EU distributor. Barbizon maybe?

Or contact them directly. They may UPS one to you.

Colin Elves
Director of Photography
Berlin/London



On 21 Jun 2018, at 18:56, Geoff Boyle <geoff.cml@...> wrote:

Which is why tungsten is the main test.

 

Unless that is someone wants us to sponsor us to go to Coober Pedy SA, Australia where we’ll probably get some good clear daylight.

 

Cheers

 

Geoff Boyle NSC FBKS

Cinematographer

Netherlands

www.gboyle.nl

Geoff Boyle
 

I’ve had updates on this and once again have to thank Nick Shaw for his help, he gets me out of all kinds of holes I dig for myself!

 

To an old man like me Cineon is a straight line, that’s what it was originally, in the dim and distant past when I learned about it. Designed for film in and then film out.

Cineon was then modified to accommodate digital imaging by adding a small toe.

Camera manufacturers have since modified that toe, and in some cases the shoulder, to make their cameras look better.

No problem with this, but do they look better in reality or just on paper?

Sure Art you can see extra stops way down on the Xyla chart but are those stops actually useable for making pictures or are they only useable in a “mine is bigger than yours ”context”?

I’ll point out again that for most cameras log means even data output per stop from around minus 4 stops all the way up to clip level.

Is the stuff bellow minus 4 important? Of course it is, but to me far less so than the areas above but then I use lights…

 

Cheers

 

Geoff Boyle NSC FBKS

Cinematographer

Netherlands

www.gboyle.nl

 

From: cml-raw-log-hdr@... <cml-raw-log-hdr@...> On Behalf Of alister@...
Sent: 21 June 2018 17:02
To: cml-raw-log-hdr@...
Subject: Re: [cml-raw-log-hdr] Further thoughts on the Camera Evaluations

 

Simple maths Art Cineon log, which is what I meant to say, allocates 90 bits per stop. Log-c allocates 80 bits per stop so is actually 12.8.

 

Except below middle grey the number of bits required reduces by 50% for each stop you go down, so the lowest/darkest stops use use far fewer bits, the darkest using a hand full of bits.  Thus it is possible to record in excess of 15 stops with using the log-c curve and 10 bit. 

 

Alister Chapman

 

DoP - Stereographer

UK Mobile +44 7711 152226

US Mobile +1(216)298-1977

 

 

www.xdcam-user.com    1.5 million hits, 100,000 visits from over 45,000 unique visitors every month!  Film and Video production techniques, reviews and news.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 



On 21 Jun 2018, at 15:36, Geoff Boyle <geoff@...> wrote:

 

Simple maths Art Cineon log, which is what I meant to say, allocates 90 bits per stop. Log-c allocates 80 bits per stop so is actually 12.8.

cheers 
Geoff Boyle NSC
EU based cinematographer
+31 637155076
www.gboyle.co.uk

 

On Thu, 21 Jun 2018, 16:32 Art Adams, <art.cml.only@...> wrote:

10 bit log-c will contain about 11.5 stops of DR and that's more than the best available domestically HDR sets can handle.

I'm curious where you got that number. I can see the full 14 stops in 10-bit ProRes. The bottom ones are pretty subtle, as they are in any camera, but they are there. I certainly don't think you lose 3.5 stops. 

 

-- 

Art Adams

Director of Photography

San Francisco Bay Area

 

 

 

 

Jeff Kreines
 

Coober Peedy!  Remember it well from about 30 years ago. 

Jeff Kreines

Sent from iPhone. 

On Jun 21, 2018, at 11:56 AM, Geoff Boyle <geoff.cml@...> wrote:

Unless that is someone wants us to sponsor us to go to Coober Pedy SA, Australia where we’ll probably get some good clear daylight.