Iron is an interesting mineral because other foods can enhance or inhibit our iron absorption. It is not enough to look at the iron content of food on a label, we need to look at our whole meal to get a sense of how iron-rich the meal is.
That said, there is some confusion about iron inhibitors — how much iron they inhibit, how bad these foods are for us to eat. The best way to think about inhibitors is to divide them into two groups: (1) Foods that contain inhibitors with little iron themselves and (2) foods with inhibitors that are also loaded with iron (iron that happens to be “locked up” in the food by the inhibitors).
Low Iron Foods With Iron Inhibitors
This category of food contains primarily beverages: milk, coffee, black tea, and red wine. These beverages typically offer minimal iron but will actually inhibit your absorption of iron in foods you eat in the same meal. Breakfast cereal is a great example: many cereal manufacturers pack your processed breakfast cereal with fortified iron because the milk you pour over it will inhibit some of that iron.
For this group of inhibitors, it is best to avoid consuming them with your iron-rich meal. I realize you are not going to give up your coffee, but you can plan an iron-rich dinner well after your early afternoon cup of coffee. You may also rely on milk for calcium like I do. If you are like me, drink the milk a couple of hours before or after your iron-rich meal. I give my son warm milk at night as a sleep aid and a calcium boost rolled into one.
High Iron Foods With Iron Inhibitors
In this category, we have high iron classics: spinach, soybeans, and corn. The iron in vegetables is blocked primarily by oxalic acid. Phytic acid is the culprit in grains, legumes, nut, and seeds. These foods tend to have enough iron that even with these iron blockers, you will still get iron out of them. They are also not so sinister that they will rob iron out of your body. I would not be afraid to eat these foods, I would just take care to prepare them to reduce their iron inhibitors. How you prepare each of these foods depends on the inhibitor.
Preparing Grains, Legumes, Nuts, and Seeds to Reduce Phytic Acid
The best way to reduce phytic acid in food comes down to one word: soak and ferment. If you place the food in warm water for hours (or even for a day), the content of phytic acid will decrease or may disappear entirely. There are some exceptions for which it is difficult to remove the phytic acid (notably, soy and corn which I will leave to other discussions). For much of the rest of the food world, you can improve the phytic acid reduction by soaking a ground version of the food (cracked or ground grains, chopped nuts). You can also improve your iron absorption by eating these foods with high vitamin C foods such as tomatoes, peppers, and oranges. There are more tricks to improving your iron absorption from high phytic acid food that I will add to this website. The tricks and exceptions are described in detail in my digital books on iron rich foods and phytic acid.
Vegetables High in Oxalic Acid
For vegetables high in oxalic acid, your best strategy is to boil them and discard the boiling water. (I describe the research in more detail here as it relates to calcium but it applies to iron as well.) Eating foods high in vitamin C along with your vegetables may help as well, especially if you are boiling all of the vitamin C out of your spinach. 
I get some flack from spinach lovers over this point and the fact is that I am a big fan of raw spinach salads but if you need the spinach for iron, your best strategy is to boil it and eat it mushy. (I know.)
Avoid Iron Inhibitors?
In general, I do not avoid foods just because they contain iron inhibitors. I am enjoying a cup of black tea as I write. However, at the times in my life when I have found myself iron deficient, I do take more care in planning my meals and avoiding iron-inhibiting foods if I cannot prepare them to reduce their phytic acid and oxalic acid. On a daily basis, I prepare much of my high phytic acid food to reduce the phytic acid. The strategies are fairly simple and, ultimately, they reduce my cooking time. It is less of a kitchen obsession than it is a regular routine.