Just ran into this when I was creating a new Custom Setting for a new org. The List Settings Type was greyed out.

Found out this is something new for Spring 18. It is now an application level settings that needs to be enabled even for System Administrators.

Starting with the Spring ’18 release, Admins have the ability to control the List type Custom Setting creation. By default, the “Manage List Custom Settings Type” is disabled, which is why Custom Setting type List is greyed out. To allow the creation of custom settings when using application-level data definitions you’ll need to enable Manage List Custom Settings Type.

To enable do the following:
Go to Setup > Data > Schema Settings and toggle button to enable “Manage List Custom Settings Type”

Then go back to Custom Settings.