The Surprising Battle to Bring the New York Death Index Online and How to Find it Free

This is the story of how Reclaim The Records fought and won the right to digitize and make available online the New York State Death Index. Thanks to their efforts these important records from 1880-1956 can now be found completely free on the Internet Archive. As Reclaim The Records points out, this story is “kind of long, but it’s really juicy, so if reading tales of government malfeasance is your thing, settle in for a treat.” We thank them for their important work and for allowing us to share this story (in their own words) with you.

Reclaim The Records, is a not-for-profit activist group of genealogists, historians, researchers, and open government advocates. They identify important genealogical records sets that ought to be in the public domain but which are being wrongly restricted by government archives, libraries, and agencies. They file Freedom of Information and Open Data requests to get that public data released back to the public. And if the government doesn’t comply, they take them to court. Then they digitize everything they win and put it all online for free, without any paywalls or usage restrictions, so that it can never be locked up again.  Learn more about their work or sign up for their newsletter.

Part I: We ask nicely

Back in January 2016, Reclaim The Records wrote and sent a simple Freedom of Information Law request to the New York State Department of Health (DOH). You can read the whole thing online if you want, because we publish all our requests and correspondence out in the open, usually through the website MuckRock, which helps people send and organize FOIA and state FOI requests.

We made two main points to the DOH in our request:

1. This 1880-1956 death index we’re asking for is already a public record, because you guys literally published for the public it years ago on microfiche and gave the copies to a few public libraries, plus the NARA branch in Manhattan. But microfiche is an old-fashioned format and hard to use, and the library copies are kind of scratched up. And a lot of genealogists with New York roots live somewhere else, and don’t want to have to travel across the country to do look-ups. Why can’t we just get a copy of that microfiche set scanned, so we can all do research at home, in our pajamas?
2. You guys at the DOH already transcribed and publish the 1957-1965 (now 1967) death index data on your own DOH open data portal, so what about the pre-1957 stuff? Why can’t genealogists get that? If all this mid-century data is okay with you, clearly not “too personal” or intrusive, then the very old data must be totally fine, right?

And so we sent off the request. And we waited to hear back.

Finally, on March 29, 2016, they gave us a response to our request. And what they had to say was shocking. Yes, the Department of Health conceded that we had the legal right to a copy of the statewide death index under the New York Freedom of Information Law. They could find no reason to legally deny our request.

So instead, they tried another way to dissuade us, by refusing to take the subsequent step in the state law, which was for the agency to estimate the actual costs of making those copies.

Part II: The state breaks the law…like, a lot

Rather than count up the actual number of microfiche sheets in question, or to provide a fair estimate for the costs of digitally scanning each of those sheets, the DOH simply decided that there would be a flat fee of $2,000 per year of microfiche copied, regardless of how many sheets were actually included in that year.

Since the microfiche set covered 1880-1956, which is seventy-six years, this meant that the DOH wanted us to pay them $152,000 for the New York State death index.

And they weren’t done yet! They also stated in this response that we had a maximum of ten business days to give them a yes or no answer to this “estimate”.

And we must then immediately provide them with a 50% downpayment, meaning $76,000. Otherwise, they said that they would close and cancel this FOIL request entirely.

(You can probably imagine what went through our head when we read their letter with these demands.)

(And you can read it for yourself, because we posted their letter to MuckRock.)

So we did what any good activist group would do, and called our attorneys at Rankin & Taylor, the same awesome bunch who had fought and won our right to the New York City marriage license index the year before.

Part III: Bring on the lawyers

Our attorneys confirmed for us that the state’s response was illegal in multiple ways. First of all, an agency that grants a FOIL request must tell you how approximately many records you’ve actually won. Here, New York was claiming that there was an unknown number of microfiche sheets per year but that the number was somehow exactly the same for each of the seventy-six years. This was absurd, as of course the number of deaths in the state varied from year to year, and grew over time as the population increased, and sometimes had unusual spikes in the death rate as happened in the 1918-1919 influenza pandemic. For the state to simply make a flat fee for each of the seventy-six years in question, even if that fee had been reasonable, was breaking the law all on its own.

But of course, that fee wasn’t reasonable, either. The cost of standard paper copies is capped in New York’s FOIL at twenty-five cents a page, but we were asking for microfiche sheets whose copying costs were not explicitly mentioned in the statue; the law just said that fees for copies of other types of non-paper records may be charged based upon the actual cost of reproduction. But the state did not provide any information on why it thought a year’s worth of microfiche scanning would cost $2,000. They did not provide any copies of estimates from vendors, nor the names of vendors, nor published price lists, nor did they ever provide some after we pressed them for details. They invented the $2,000 per year cost, and then never explained or apologized for it.

Thanks to a genealogist friend who happened to be in Albany and who took it upon herself to count up all the microfiche sheets of the death index at the library, we knew there were 1,522 microfiche sheets in total in the state death index, give or take the few sheets that had gone astray over the years.

Based on quotations from multiple digitization companies whose price lists we found online, we were able to establish that the actual costs of scanning each microfiche sheet would be about $2 per sheet, maybe a little less if we were doing a big bulk order. That meant that the actual cost of fulfilling this FOIL request should have been about $3,044 dollars for the full seventy-six years of microfiche scanning, plus a little more for the cost of shipping.

Even the National Archives (NARA), whose prices tend to be some of the highest amongst archives who directly create scans and copies for researchers, say they will prepare microfiche scans for researchers for $12 per sheet, although they also include free shipping. Even at that inflated cost, our death index project would have cost about $18,264 in total – expensive, yes, but still far, far less than Albany’s “estimate” of $152,000! (For the record, the current median price of a home in Albany is $166,200.)

Finally, our attorneys confirmed for us that the Department of Health’s demand that we provide them a 50% downpayment for their made-up $152,000 estimate within ten business days – a demand known as an “exploding offer” – was entirely illegal. There is no such requirement in New York State’s FOIL that a records request be agreed to in a specific timeframe, much less in one so short a window. The Department’s Records Access Officer had completely made it up. And they never apologized for it, even after they dropped that part of the demand when our attorney called them out on their behavior.

Our attorney also set up a phone conversation with the pro bono New York State Committee on Open Government, a watchdog group funded by the New York legislature, and their executive director confirmed for us that the Department of Health was breaking FOIL and acting in bad faith. Furthermore, the Department’s Records Access Officer had exceeded a reasonable or legal timeframe for even making this wholly ridiculous response. They had responded 143 days after our original request, and fifty-three days after we had repeatedly asked them to justify their outrageous fees; New York’s FOIL requires a full government response in five business days and provides for a twenty business day limit in unusual circumstances.

This kind of legal trickery, first a delay and then a refusal to provide a real estimate of the records’ reproduction costs, was insidious, because on the surface, the state could act innocent and claim that they hadn’t really blocked off our legal access to the records, it was just that we had been unable or unwilling to pay for them. It was essentially a non-denial denial, or a “constructive denial” as the legal terminology calls it – but getting a Department of Health watchdog to side with us instead of their own agency would be difficult. And that’s just what happened.

For reasons that defy explanation, when we filed a formal appeal of this FOIL request with the Department of Health on June 3, 2016, the person to whom we appealed, who also worked for the Department, proceeded to deny our appeal.

Part IV: FOIL request, take two

At that point, we had a choice to make. We could take the next step and take the case to trial, suing the state of New York. We would probably win, but it would be expensive, and there was no guarantee that we would win back attorneys fees even if we did win all the records. In New York, the awarding of attorneys fees is left up to the discretion of the judge, and is a separate question from whether the records were wrongly being withheld.

But we had another option. In all the months of fighting with the DOH for these records, our point person on the case had been their Records Access Officer, a woman whose name we will omit here but which can be easily determined from reading through our legal paperwork, which are all a matter of public record and all online. She had been the person behind the lack of transparency on pricing, the ten-day exploding offer, the months of delays, and all of that.

And as we were approaching the legal deadline of whether or not to pursue legal action against the state, we heard through the grapevine that this woman was suddenly no longer the Records Access Officer. We hadn’t yet had time to file a formal legal complaint against her and her office’s actions; perhaps she had gotten into trouble for unethical and illegal behavior with other records requestors? In any case, she was now gone.

This meant that instead of suing the state, we could try a different tactic. We could re-file a new FOIL request for the death index, starting from scratch, with the new Records Access Officer, hoping that this one would be more reasonable than her predecessor. And that’s just what we did. Luckily, this new one was far more pleasant to deal with, and as far as we could see she did not do anything overtly illegal.

Part V: Can we scan that for you?

At this point, it was nearing the end of 2016. In an effort to move things along, we asked the new Records Access Officer if we could bring in our own digitization vendor to do the scanning of the microfiche sheets they held, just to make it easier on the state and get these records copied already. After all, they’d already said on their own letterhead that we were legally entitled to a copy of the files; we were now basically just nitpicking over how much it would cost and who would pay for it.

She replied that we could indeed hire our own vendor to do the scanning…with a few minor restrictions. First of all, our vendor would have to come onsite to Albany, and could not remove the sheets from the state facility. Second, the vendor would only be allowed to do their work there during business hours on weekdays and when employees of the Department had free time to supervise them doing the work directly. Third, they could only do the microfiche scanning sheet by sheet, hand-fed into the machine instead of using the automatic feeder, a process that would take much longer, and therefore incur a higher cost. Fourth, we had to provide a full spec list and product manual of the brand and exact model of the scanner that our vendor would use, in order to get the approval from the state. And so on.

Frustrated but persistent, we said yes to all these demands. Yes, we’ll agree to your picky timetable; yes, we’ll get you all the details on our magic microfiche scanner. But can we just have the death index already? Please?

Part VI: New York hires an archivist

Well, no. Because the next hoop the state wanted us to jump through was this: we were told that we now had to wait a few more months while the state hired an archivist to come to Albany and assess the quality of the microfiche copies of the death index they held in their state vaults, to see if they were too fragile to scan with this magic microfiche machine. And that meant that we also had to wait while they discussed and approved the government funding to hire this archivist.

So while we waited, we tried yet another tack to speed up this request, which was by now more than a year old. Hey, remember how there are about eleven libraries in New York State where the public can use a copy of this microfiche? Sure, those copies of the microfiche are in much worse physical shape than the ones in the Department’s vaults, but they’re available right now. Can we maybe scan one of those copies instead of waiting for you guys in Albany to get an archivist to assess your copies? Wouldn’t that be faster and easier?

Part VII: Nope, not ours

And here’s one of the nuttiest things that happened in this whole story: the New York State Department of Health then repeatedly denied to us that these copies of their own microfiche, which are in libraries across the state through a strict legal arrangement between the Department and those libraries, ever belonged to them in the first place. They claimed that the libraries are under the Department of Education, and to get access to their holdings, including these death index microfiche, we would have to file a FOIL request with the Department of Education instead of the Department of Health.

In other words, we were the ones who had to inform the Department of Health about their own property, their own microfiche copies of the statewide death index, being available throughout the state. And even when we did, they repeatedly denied to us in print and by phone that these microfiche in their libraries are their property. So we decided not to pursue the idea of duplicating the libraries’ copies of the fiche, and to wait out the arrival of the new archivist to confirm that the state’s own copies were in acceptable condition.

Part VIII: Free at last?

And then? Another bombshell dropped – but this time, a good one, or so it seemed, The Records Access Officer, on one of our many phone calls to her asking for a status update, suddenly let us know that the Department would be hiring a vendor to scan the vault copies. And the state would not charge us anything for the vendor’s labor. They didn’t even give us an estimated cost at all.

This was wonderful news, but also quite odd – hadn’t we just been spending months with the previous Records Access Officer fighting over the cost of the scanning? But now it was all going to be handed to us for free? We didn’t want to look a gift horse in the mouth, so we didn’t ask too many questions. And a few months later, seventeen months after the original FOIL request had been filed, we finally received in the mail a small USB hard drive containing all the seventy-six years of scanned microfiche. It was over, Reclaim The Records had the New York State death index.

Part IX: There’s no such thing as a free lunch

Several months later, we found out why Reclaim The Records had never received a bill, or even an estimate, from the state itself or from the mysterious vendor who had scanned the microfiche sheets on behalf on the state.

The vendor turned out to be none other than Ancestry.com, the world’s largest for-profit genealogy company. They, along with every other genealogy organization in the world, as well as those from many other industries like insurance companies, had been hammering on New York State for years, for decades, begging for copies of the incredibly valuable data of this statewide death index. But not one of these companies or organizations had ever cracked the wall…until Reclaim The Records showed up on the scene with our awesome new approach of using Freedom of Information laws to force the state’s hand.

And at that point, Ancestry apparently felt threatened and did not want to be cut out of the records pipeline entirely. Even though Reclaim The Records always has and always will post all our data online for free, whether images or text, clearly in the public domain, it seems like Ancestry wanted to make sure they still were a player in the game with unique access to the state. And so unbeknownst to us, they had gone to the state and said look, Reclaim The Records is gunning for you guys, they are clearly going to win their legal case against you, how about you let us at least scan the data for you for free?

On the one hand, this was a very generous thing for Ancestry to do, as it meant that our little non-profit organization didn’t have to bear the cost of thousands of dollars for scanning, which the state was certainly allowed to charge us under FOIL. But there was also an ulterior motive: by volunteering to be the vendor doing the actual scanning, Ancestry was guaranteeing that they had access to the raw images of the statewide death index before anyone else, even before the state itself. This meant that they had a head start on scanning and perhaps even indexing this brand new data and preparing it for upload to their website, while we at Reclaim The Records, the original FOIL requestor, waited ignorantly and patiently for our own copy to arrive in the mail.

It also meant that a few months later, when Ancestry had finally transcribed the entire New York State death index and revealed it on their website as a fait accompli, they put their copy of both the images and the new searchable database behind their paywall. That would have been fine with us, except for one little issue: they never once mentioned how they acquired that data in the first place. They presented it entirely as their own achievement.

Part X: Ancestry takes the credit

Ancestry never mentioned that Reclaim The Records had initiated the FOIL request, had fought the state and their underhanded tactics for over a year, had paid our own attorneys for legal help. They never mentioned how Ancestry had worked a backroom deal with New York State to be the first-ever scanning operation of these files, all just to get a few months’ headway on this public data. And in their publication of the new database, including their press releases and blog posts and even a specially-made video introduction they filmed for social media, they never once mentioned a single word about Reclaim The Records, or our work, or gave a link to our website.

Ancestry will, however, make quite a bit of money off these millions of valuable New York records being posted behind their paywall, an especially important concern as news reports indicate that the company was until recently moving towards an IPO. Their estimated valuation in 2016 was $2.6 billion; Reclaim The Records’ actual valuation in 2016, at the time of our initial FOIL request, was literally zero dollars. We did it for the love of open public records access.

Was all this legal? Sure. We still had our copies of the data we’d won. Ancestry’s copies of the images, being the original ones created directly from the state’s films, technically didn’t come from us, so no credit was required. But was it right to cut all of our work and initiative out of the story? Was it ethical, or even just plain nice?

We will leave this question as an exercise for the reader.

Part XI: We share our toys

Meanwhile, Reclaim The Records took our hard-won copies of the statewide death index and returned our copies to the public domain, totally free, just as planned.

We uploaded everything to the Internet Archive, where people can flip through the pages of each year’s films for free, without any paywalls or logins. Most of the files are in good to excellent quality, except for a few years from the 1940’s, where even the state’s vault copies had deteriorated to a terrible degree, showing how critical it was that an organization like us had forced New York State to finally make a digital copy and preserve their own data.

We’ve also reached out to other non-profit genealogical organizations, including to regional groups like the New York Genealogical and Biographical Society, and the New England Historic Genealogical Society, giving them an early heads up about the impending release of the records and asking them if they’d like to host copies on their own websites. Both groups have enthusiastically agreed, and will add the material to their existing data collections in the near future. We’re also thrilled to hear that other genealogical organizations like FamilySearch and MyHeritage will be making their own transcription projects for these images. Eventually there will be free copies of a text database of the New York State death index, not behind paywalls, for everyone to use.

Part XII: We FOIL Albany, Buffalo, and Yonkers

There are still a few loose ends to wrap up. The statewide death index was missing deaths for three cities for several decades; Albany, Buffalo, and Yonkers did not cooperate with the statewide registration requirements until about 1915, and their deaths are mostly missing from the index prior to then.

So, just a few days ago, Reclaim The Records filed three brand new FOIL requests with the city clerks’ offices for these three cities, asking for copies of their death indices prior to 1915. The acquisition and publication of that missing data may take another year to complete, depending on whether or not we will need to initiate legal action against one, some, or all of three of the agencies for non-compliance with the law.

You can read all the back-and-forth discussion with the three agencies here, at the MuckRock pages for our new requests: Albany, Buffalo, Yonkers.  (The Buffalo e-mails make for especially fun reading.)

Part XIII: MORE AWESOME RECORDS REAL SOON

So that’s the story of the New York State death index, and how we won it. Phew!

And if you’ve read this far, here’s a sneak peek of a records set that we’re about to announce in the next few weeks: it’s 116 years of brand new vital records data from a northeastern state! (No, it’s not New York this time.) Couple of million names. You’ll like it.

Do you know of records that should be digitized and placed online for free but that are currently being locked away? Post them in the comments. You can also contact Reclaim The Records or make a donation through their website.

Image: DiCesare/Corona monument in Holy Sepulchre Cemetery, Rochester, NY. Einar Einarsson Kvaran. GNU Free Documentation License