Include flags in building C++


#1

@seantalts and I were talking this morning. I found that the -isystem flag and the -I flag do different things for includes.

Here’s how to reproduce and see the difference:

Using -isystem for libraries

clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wsign-compare      -O3 -I . -isystem lib/eigen_3.3.3 -isystem lib/boost_1.66.0 -I lib/sundials_3.1.0/include -isystem lib/gtest_1.7.0/include -isystem lib/gtest_1.7.0     -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION    -DGTEST_USE_OWN_TR1_TUPLE  -c -E -M -MG test/unit/math_include_test.cpp -o dependencies.isystem.txt

Using -I for libraries

clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wsign-compare      -O3 -I . -I lib/eigen_3.3.3 -I lib/boost_1.66.0 -I lib/sundials_3.1.0/include -isystem lib/gtest_1.7.0/include -isystem lib/gtest_1.7.0     -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION    -DGTEST_USE_OWN_TR1_TUPLE  -c -E -M -MG test/unit/math_include_test.cpp -o dependencies.I.txt

There’s a bunch of stuff that gets included then…

  stan/math/rev/core/chainable_alloc.hpp \
  stan/math/rev/core/chainablestack.hpp stan/math/rev/core/ddv_vari.hpp \
  stan/math/rev/core/vari.hpp stan/math/rev/core/dv_vari.hpp \
  stan/math/rev/core/dvd_vari.hpp stan/math/rev/core/dvv_vari.hpp \
  stan/math/rev/core/empty_nested.hpp \
  stan/math/rev/core/gevv_vvv_vari.hpp stan/math/rev/core/var.hpp \
  stan/math/rev/core/grad.hpp stan/math/rev/core/nested_size.hpp \
  /usr/local/include/boost/math/tools/config.hpp \
  /usr/local/include/boost/config.hpp \
  /usr/local/include/boost/config/user.hpp \
  /usr/local/include/boost/config/detail/select_compiler_config.hpp \
  /usr/local/include/boost/config/compiler/clang.hpp \
  /usr/local/include/boost/config/detail/select_stdlib_config.hpp \

vs

 stan/math/rev/core/chainable_alloc.hpp \
  stan/math/rev/core/chainablestack.hpp stan/math/rev/core/ddv_vari.hpp \
  stan/math/rev/core/vari.hpp stan/math/rev/core/dv_vari.hpp \
  stan/math/rev/core/dvd_vari.hpp stan/math/rev/core/dvv_vari.hpp \
  stan/math/rev/core/empty_nested.hpp \
  stan/math/rev/core/gevv_vvv_vari.hpp stan/math/rev/core/var.hpp \
  stan/math/rev/core/grad.hpp stan/math/rev/core/nested_size.hpp \
  lib/boost_1.66.0/boost/math/tools/config.hpp \
  lib/boost_1.66.0/boost/config.hpp \
  lib/boost_1.66.0/boost/config/user.hpp \
  lib/boost_1.66.0/boost/config/detail/select_compiler_config.hpp \
  lib/boost_1.66.0/boost/config/compiler/clang.hpp \
  lib/boost_1.66.0/boost/config/detail/select_stdlib_config.hpp \

#2

I’m so confused - how can make know that “-isystem lib/boost_1.66.0” should be replaced with “/usr/local/include/boost”?


#3

This is expected behavior. ‘-isystem’ places our boost ahead of system boost so our boost is found first. ‘-I’ places our after so you get system boost. It’s possible that if we’re using a boost header that’s missing from our boost then it gets picked up from system boost which could explain that weird Ubuntu failure!


#4

Maybe I have the order mixed up in what @syclik is claiming, then it would be confusing.


#5

Ummm… make only knows what we set. It’s a matter of whether we include our libraries using -isystem or using -I.

The compiler needs to know what it’s including. That call that generates dependencies doesn’t have anything to do with make.

Yeah… it’s backwards. -I includes it before the system directories. -isystem includes it after (it will pick up the system directories first).

Here’s the GCC “Options for Directory Search” documentation. It really doesn’t say anything about that behavior. Here’s why I thought -isystem was supposed to be the right thing to use:

You can use -I to override a system header file, substituting your own version, since these directories are searched before the standard system header file directories. However, you should not use this option to add directories that contain vendor-supplied system header files; use -isystem for that.

I interpreted our use of libraries as “vendor-supplied system header files.” But I think the thing that’s really misleading is that there’s no indication in the documentation that the include order should change by using -I or -isystem.


#6

I think I see what they mean. This stackoverflow post has a clarifying quote:

Add the directory dir to the head of the list of directories to be searched for header files. This can be used to override a system header file, substituting your own version, since these directories are searched before the system header file directories. However, you should not use this option to add directories that contain vendor-supplied system header files (use -isystem for that). If you use more than one -I option, the directories are scanned in left-to-right order; the standard system directories come after.
If a standard system include directory, or a directory specified with -isystem, is also specified with -I, the -I option will be ignored. The directory will still be searched but as a system directory at its normal position in the system include chain. This is to ensure that GCC’s procedure to fix buggy system headers and the ordering for the include_next directive are not inadvertently changed. If you really need to change the search order for system directories, use the -nostdinc and/or -isystem options.

I think by vendor-supplied they mean your distribution supplied them?

I guess we probably do want to override any locally installed boosts, right? Though I’ve seen people ask for the opposite sometimes, they can set that in make/local with BOOST= I believe. So it sounds like we should try switching to -I and just pray that we don’t inherit compiler warnings (or maybe change the stance on that / figure out other ways to suppress them?).


#7

The gcc man page (at least on arch linux) is informative about how this all works, search for -isystem and (gah) -idirafter

Yeah, I got that wrong, it really looks like we should be using -I throughout and possibly -isysroot if things get really hairy. I’d look into this but I have two other Stan things promised already that are getting done first :P

yeah, in general, agree it would be nice to say what we override it with.

I’m still concerned that if our version of Boost is missing a header it will get mixed in from the system install unless we play games with -isysroot… of course playing those games would mean downstream projects need to set their own flags (can’t imagine R’s build system would like that option).


#8

Mind quoting it or finding an online version? That sounds really useful – I’ve never found doc that describes what order things are included.

Yeah, I think we’ll want to use -I

We will inherit compiler warnings. I think we just need to suppress them with compiler options. It sucks, but if we can’t rely on the use of -isystem, then I think that’s a reasonable option. If you figure out a way to selectively suppress warnings, that would be awesome. I’ve never seen it, but maybe there’s a different option that I don’t know about.


#9

Sure (attached), search for the -isystem section. gcc-archlinux-man-page.txt (1.0 MB)


#10

Thanks much. The behavior contradicts the documentation.

Here’s the relevant part:

       You can specify any number or combination of these options on the command line to search for header files in several directories.  The lookup order is as
      follows:

      1.  For the quote form of the include directive, the directory of the current file is searched first.

      2.  For the quote form of the include directive, the directories specified by -iquote options are searched in left-to-right order, as they appear on the
          command line.

      3.  Directories specified with -I options are scanned in left-to-right order.

      4.  Directories specified with -isystem options are scanned in left-to-right order.

      5.  Standard system directories are scanned.

      6.  Directories specified with -idirafter options are scanned in left-to-right order.

The part that’s wrong is that it sounds like instead of (3, 4, 5), in implementation it’s (3, 5, 4). The -isystem directories are after the standard system directories in the lookup order.

Mind double-checking that I read that doc correctly?


#11

Yeah, you read the doc right, I’d have to do more checking to really understand what order things are happening in for real (in implementation). I don’t think we should worry about the warnings, just capture the compiler output wholesale and go from there. … even in rstan we should be able to do it.