Topics

Lens Performance Chart Tool - Survey

William Griffiths
 

Hello,

I am a third year science university student currently writing a technological report regarding how camera lenses differ in quality.


The aim of the report is to test a select number of prime lenses and create a simple score-based performance chart that allows users to quickly and easily compare various lens attributes, such as distortion and focus breathing, between lenses so that they can choose a lens that's right for the job. Ideally, a fully developed interactive version of this tool would be presented as an app or website.

 

I have designed a prototype of this tool, a simple proof of concept created in Excel, and I would like to get some feedback in order to improve the design and gauge whether this type of tool is desirable to industry professionals.

 

If you have time and interest, it would be greatly appreciated if you could please fill out this quick survey to provide feedback: https://goo.gl/forms/lS0CzqEd7T6L0umE2

 

All respondents will be kept anonymous with only their job title disclosed.

The performance charts and more information regarding them can be found within this document (Word and PDF versions available): https://drive.google.com/open?id=1y4jKKgiQZAS7P7el-lerC1JhOAmctQJJ

Please ensure you are using WiFi as there are images and documents in the links.

 

I would like to thank Geoff Boyle for granting me permission to post this on CML and for taking the time to discuss this project with me.

 

Thank you for taking the time to read this and for completing the survey.

 

William Griffiths

BSc (Hons) Film Technology and Visual Effects – Third Year

Birmingham City University

Birmingham, UK

Matthew Duclos
 

Bravo! Very well done. I recently presented a similar chart during a speaking engagement for Fujinon which was affectionately dubbed the Chrysler Diagram (you know... that company that used to make cars). I'll be sure to submit feedback via the survey. One tip: I would recommend cutting out as many variables as possible. For example, how long do you plan to use a 5D as your test platform? As soon as you switch cameras, your test results can vary dramatically since a specific sensor can effect the performance of a given lens. I've tested thousands upon thousands of lenses in my life, 99% of which I've done on the exact same test projector using the exact same test reticle. The key to relevant lens comparison tests is consistency. Best of luck in your endeavors! 

--
- Matthew Duclos
Lens Tech/Geek
Los Angeles, CA

niclaw@...
 

Hi William,

generally your research and observations are neutral and allow the reader the opportunity to come to their own conclusions as to the 'value' of any given lens feature. However, the key to your charts makes a questionable statement when you allocate the terms "Excellent Quality" and "Poor Quality" to the different ends of the scale. You mention this factor in your writing, but it doesn't seem enough to mention it and then ignore it.

Geoff's recent lens tests show that we all value different features according to personal preference and relevance to the desired outcome. As a result, a lens that breathes or is soft may be an 'excellent' lens for the outcome required. On that basis I would suggest you go with something a little more prosaic for your key, such as simply "High" or "Low" in which case you might need to invert some of your scales. This would leave you with charts that may be more complex to interpret, but at least don't make a claim to know what the cinematographer/photographer is trying to achieve.

I can imagine an app like this being useful, but it seems apps aimed purely at cinematographers are difficult to fund and maintain.

I wish you the best of luck,

Nic Lawson
DP London UK

William Griffiths
 

Matthew Duclos,


Many thanks for the positive response, it's very reassuring to say the least!


It’s interesting to see you presented a similar chart so it would seem that this type of tool is something that some would like to see.

Regarding the cameras, the idea for this is to see how lenses perform on a variety of cameras, preferably as many lens and camera combinations as possible, much like DxOMark. So I suppose it would be more accurate to refer to this as an ‘Imaging System Performance Chart’. Personally I feel it is important to see how lenses perform on a variety of platforms since camera choice can drastically affect performance, particularly on crop sensors where aberrations may be lessened in an image since aberrations typically worsen at the edges of a lens among other changes in performance. However, the purpose of the survey is to see what could be improved, and keeping the camera consistent may indeed be something others would like to see.


Again, thank you so much for the response, I’ll be sure to take note of all feedback when evaluating this in my report. I may also share the finished report if it gets published and if there is an interest in reading it.

 

William Griffiths

BSc (Hons) Film Technology and Visual Effects – Third Year

Birmingham City University

Birmingham, UK

George Hupka
 

I caught some of Matt's NAB presentation and that chart structure also jumped out at me!

Couldn't agree more about cameras affecting the tests.

I wonder if it's possible to use a lens projector (eliminating the camera as a variable) but accounting for different image circles (sensor sizes)? There are still a lot of various sensor sizes, but barring testing dozens of cameras, it may be possible to give "camera neutral" results for a variety of sensor sizes.  Barring that, it's a question of how many cameras - and formats for each camera - you can practically test.

Other factors that affect my choice of lens include close focus distance and size/weight - "qualities" that may not be measurable on a chart but certainly affect the choice of lens for different applications.  That may be a bit off the subject of things that are measurable with a test chart - but are relevant to what makes a "well rounded lens" as Matt discussed in his presentation.  Items like this are listed in your table at the bottom - is the idea to allow the user to select any of those items to include in the graph?

If so that would be brilliant - if I'm doing a project where I know I need lightweight lenses I can see the trade-offs I'm making in other performance categories.  For any specific project I might be able to live with one compromise but not another...

Lots of work for you with this project!!!

--
----------
George Hupka
Director/DP
Saskatoon, Canada
Listmum, Cinematography Mailing List

Geoff Boyle
 

I missed this message and had to drag it out of my spam bin to see what people were commenting on, it wasn’t in my spam bin.

It was in my student folder as William is one of the students I have direct contact with through my sessions at BCU!

 

I agree with Matthew about the camera having an effect on the results but hey! He’s a student and has limited resources.

 

I’d love to have an app that showed breathing etc.

 

Cheers

 

Geoff Boyle NSC FBKS

Cinematographer

Zoetermeer

www.gboyle.co.uk

+31 (0) 637 155 076

 

From: William Griffiths <william.griffiths@...>
Sent: 15 April 2018 16:13
To: geoff.cml@...
Subject: [glass] Lens Performance Chart Tool - Survey

 

Hello,

I am a third year science university student currently writing a technological report regarding how camera lenses differ in quality.


The aim of the report is to test a select number of prime lenses and create a simple score-based performance chart that allows users to quickly and easily compare various lens attributes, such as distortion and focus breathing, between lenses so that they can choose a lens that's right for the job. Ideally, a fully developed interactive version of this tool would be presented as an app or website.

 

I have designed a prototype of this tool, a simple proof of concept created in Excel, and I would like to get some feedback in order to improve the design and gauge whether this type of tool is desirable to industry professionals.

 

If you have time and interest, it would be greatly appreciated if you could please fill out this quick survey to provide feedback: https://goo.gl/forms/lS0CzqEd7T6L0umE2

Brian Rose
 

Couldn't agree more about cameras affecting the tests.

 

Although a projector will indeed overcome the variations introduced by both cameras and individual setups, it is somewhat restrictive. It’s fine for measuring resolution in a single image plane, as well as other aberrations such as geometric distortion, coma, chromatic etc. But it won’t show what’s happening in the three dimensional world. As David Samuelson used to say,  you can use flat charts, but how many people make movies of flat charts.

There are probably more interactions taking place with today’s digital cameras as there were with film cameras; Boka, the Nacissus Effect, the interaction with the low pass filter, etc.

So it’s always the lens/camera combination. I used to think that we could indeed carry out all our tests with charts and projectors. Whilst the second was great for overall testing and finessing lenses such as the Excellence Periscope, in the end, it was the camera tests that really showed us what was happening in the real world.

 

 

 

Brian Rose

Former Technical Manager OpTex Broadcast Rentals
Tel: +44 (0)20 8868 1729
Mob: +44 (0)7768 635 788
email: brianmrose@...

 

John Brawley
 

I applaud the intent behind this.

But it seems to me that it can only be possible to measure some properties of a lens.  Others seem to be hard to define.  Like the way they model or render a face. Is this Geometry ? 3D-ness ?

I feel like the measurable metrics like CA or resolution are only part of the story.

Then there are the subjective or taste metrics, which Geoff’s latest blind tests show are really very difficult to define and quantify.  How do we account for those ?

JB

John Brawley
DP - currently Chicago Illinois

William Griffiths
 

I’m going to address some things from this thread.


Firstly, Matthew Duclos, would you be able to disclose any information regarding the Chrysler Diagram you presented? It is okay if not, but it would be good for my report to include how others such as yourself feel that a chart such as this may be of use in the industry – any information at all would be greatly appreciated.

George Hupka, when designing this I believed that it may be best to leave specifications such as weight and cost as a simple list considering those are quite specific. On the other hand, if you believe adding these attributes to the chart in some way, I’m sure there may be a way to include those either on the main chart or in a separate one that has an open scale since lenses have a relatively large range of price, weight, etc. But, yes, as you say, the purpose of the chart is to demonstrate trade-offs and compromises in a range of attributes to pick what lens may suit a certain job. For instance, a photographer probably would pay no mind to focus breathing because it does not affect their work as much as a Cinematographer’s so they can afford to pick a lens with higher focus breathing but has more desirable attributes than a lens with less breathing.

 


With reference to John Brawley, the way lenses render faces would be attributed to perspective distortion considering it affects how background elements appear relative to the foreground, which affects things such as a face albeit to a lesser extent – regular distortion may factor into it as well.

I do also agree that there are some things such as lens flare (Geoff Boyle) and bokeh that are usually looked for in lenses but are far more subjective and difficult to quantifiably measure than aberrations such as distortion or CA which are relatively simple to determine. As demonstrated in an alternative design in the survey (which seems more popular than the main scoring system), a minimum/maximum effect score may be able to address this, particularly bokeh although I am unsure as to how one would determine lens flare considering there are many factors that affect how a flare looks such as radius, colour and length – it’s not as clear-cut as minimum/maximum effect, therefore it is quite difficult to show other than displaying images or footage of a lens' lens flare.

 

Many thanks to those who have responded to the thread and/or completed the survey thus far, it has been an interesting response. And just a reminder that it would be helpful if you could pass this along to other professionals who use cameras and lenses on a regular basis to help increase the survey sample size.

 

Thanks again,

 

William Griffiths

BSc (Hons) Film Technology and Visual Effects – Third Year

Birmingham City University

Birmingham, UK

George Hupka
 

"George Hupka, when designing this I believed that it may be best to leave specifications such as weight and cost as a simple list considering those are quite specific. On the other hand, if you believe adding these attributes to the chart in some way, I’m sure there may be a way to include those either on the main chart or in a separate one that has an open scale since lenses have a relatively large range of price, weight, etc. But, yes, as you say, the purpose of the chart is to demonstrate trade-offs and compromises in a range of attributes to pick what lens may suit a certain job. For instance, a photographer probably would pay no mind to focus breathing because it does not affect their work as much as a Cinematographer’s so they can afford to pick a lens with higher focus breathing but has more desirable attributes than a lens with less breathing."

Cost and size/weight were two of the axes on Matt's "Chrysler" chart at NAB, which is what got me thinking of these and other factors that could be included. 

You obviously might not want to have every graph with 18 axes, but... If I'm looking for certain lens characteristics, it would be great to select those characteristics to include in the graph, so that whatever is important for any given project is included in the graph.  For example, if I need a lens with excellent sharpness for shooting VFX plates, I can easily see what the trade-offs are... (usually size/weight and cost!) maybe one of the newer budget lens manufacturers has a product that would fit in perfectly for my needs at a much lower cost.   Sure, it might breathe a lot, but if I'm shooting plates that's not a problem.  Or it might not be fast, but if my plates are all T8 outdoors, again, not a problem.   Seeing those factors represented visually can definitely make it easier to narrow down your choices... or realize that there's a lens that matches your needs that you would never have considered.

I would see this as a tool that helps winnow down the ever-growing selection of lenses - especially lower cost lenses that have more trade-offs - so that I can decide which lenses I want to look at and test for a particular project.

--
----------
George Hupka
Director/DP
Saskatoon, Canada
(Currently in New York, NY)
Listmum, Cinematography Mailing List