In prior posts here and here, I analyzed new data obtained from FDA through the Freedom of Information Act about FOIA requests.  I looked at response times and then started to dive into the topics that requesters were asking about.  This is the third and final post on this data set, and it builds on the last post by taking the topics identified there to explore success rates by topic.  From there, I look at who is asking about those topics and how successful those individual companies are in their requests.

Reviewing the topics

In my April post, I laid out topics that I obtained through natural language processing.  I won’t repeat the analysis about how I arrived at the topics, but I just want to remind you what those topics are.

Word Count and Importance of Topic Keywords

As I explained last month, some of those topics are easier to understand and interpret than others.  The ones that seem to fall into somewhat defined categories include the following:

• 2. Complaint Investigation/Adverse Events
• 3. Drug Inspections
• 4. 483/EIR Drug
• 6. NDA/Adverse Event Report
• 7. Warning Letters
• 9. Specific Correspondence
• 11. Follow-On FOIA
• 12. 510(K)
• 13. Indian 483
• 17. Meeting Minutes
• 18. Medical Device Inspection

I would encourage you, however, to look at the table above and especially look at the topics where I haven’t labeled them, because they might be meaningful to you from your own perspective.  For those topics that I didn’t think lend themselves to a pithy title, I simply continue to refer to them by number.

Chances of Success by Topic

Given those topics, I wanted to understand which topics are likely to succeed in getting information from FDA.  Here’s what I found.

I added the column at the right to show you the lengths of the blue bar as a percent of the overall length of the total bar.  While you can roughly eyeball it, if you’re like me, the numbers make it a bit more specific and concrete.  Thus, for example, for the FOIA topic of a “483/Adverse Event Report,” over all the requests for the last 10 years, requesters are 69% likely to get a “closed” response, which means FDA sends responsive documents.

Who is asking what?

I was then interested in exploring at a deeper level which particular organizations were requesting these particular topics.  The best way to present the high-level information here is to take the top 20 requesters and use a heat map to explore the relationship between the requesters and the topics.  The deeper the color, the more the requester is asking about those topics at the intersection of the requester and topic lines.

So, for example, it’s apparent that FDAZILLA submitted a whole lot of topic number 10 requests.  You can read about the words that defined topic 10 in the table above.  On the other hand, the white spots indicate areas where particular firms didn’t submit any significant number of requests.

Who is successful?

The final question I wanted to explore was success rates for different requesters.  The following chart shows the number of requests a given requester made, while the colors indicate the outcome of the request.

FOI Services submitted a lot of requests, but their success rate was proportionately not as good as FDA News, for example.

Methodology

Most of the work was done in advance of the prior posts and summarized there so I won’t repeat it here.  For this post, I needed to simply use some different Python libraries like seaborn to produce the heat map visualization.

Interpretation

The chances of success in a FOIA request apparently depend heavily on the topic generally.  It should not be any surprise that a follow-on request is highly successful, 80%.  That’s because all of the hard-work and debate went into the original request.  Of course, it’s possible for people to file a follow-on request before the earlier request has even been resolved, so those follow-on requests might account for at least part of the 20% that did not succeed.

Getting a particularly low score is meeting minutes, which only succeeds about 42% of the time.  That’s not surprising, because typically someone would probably ask for meeting minutes for a sensitive topic that might reflect either ongoing policymaking or enforcement efforts.  Those requests would be rejected as exempt from disclosure under FOIA.

Warning letter requests don’t fare much better at 54%.  Obviously, people are not asking for the warning letter itself as that is publicly released already.  They are apparently asking for enforcement related documents that led up to the warning letter.  Many of those, if requested soon after the warning letter, would be exempt from release as part of a not yet completed enforcement action.

Requests related to 510(k)’s have a pretty high success rate at 73%, but as my first post indicated, you need to be really patient when you request that.  It can take a couple of years.

The lowest success rate relates to what looks like a somewhat specific request – topic 19– that somehow involves pet food and emails about it, as well as the Association of American Feed Control Officials (AAFCO).  I didn’t spend any real time researching it, but I would guess that there was some controversy about a pet food regulatory model developed by AAFCO that requesters wanted information about.  Those FOIA requests were only successful 37% of the time.

I think it’s interesting to then see who are the ones asking about these various topics, and what their individual company success rates are.  As you can see from the heat chart, Merck likes to submit follow-on FOIA requests, and when you look at the chart about company success rates, Merck is very successful in it.  That makes sense.  As already noted, follow-on FOIA requests are among the most successful.

FOI Services, on the other hand, likes to ask for a lot of 510(k)’s.  But they also asked for just about everything else.  Overall, their success rate is much more mixed.

FDA news, which does more investigative reporting, was very interested in the Indian 483’s as well as 483’s more generally.  What I find interesting is that they have a very high success rate, despite the fact that those individual topics do not have as high a success rate generally.

Conclusions

Your success in getting information from FDA is, not surprisingly, tied to the topic about which you are requesting.  But it also seems to be tied to who you are.  Or maybe it’s less about who you are, and more about how competent you are.  Not all requests are equally well written.  Nor are they equally well-timed.  People who ask for information prematurely, before the agency is done with its confidential work, are more likely to be unsuccessful in getting it.  There are a lot of factors that go into whether a request is successful, even if it’s on a very similar topic.

Part of my purpose in doing this was to be on the lookout for any suggestion that somehow someone was effectively gaming the system and getting responses that they should not get.  It’s hard to assess that looking at such big data volumes because, for example, I really only looked at the top 20 requesters.  There are simply too many requesters to do a detailed review of all of them.  But at least among the big ones, I was comforted in that it seemed as though the data were all rational to me, suggesting that the system is working as intended reasonably well. The only thing that really bothered me in all three of these posts is the extraordinarily long tail in the response times.  As I pointed out at the end of my first post, about 12% of the time, resolution requires more than 400 days, over 13 months.  More than that, the tail is extremely long in that some requests actually go on for many years, up to about 10 years.  At that point, I think you have to say the system is broken because for nearly anyone, whether media, commercial, or academic, waiting over a year let alone 10 years renders the information often useless.  There must be some way that we can keep requests from going past a year.  However, I imagine fixing that will require not just FDA but Congress.