Hemp archive: 30 years now digitized

Since the end of the 1980s, activists have been collecting information on the subject of hemp, mainly from Switzerland, but also internationally. The focus is on political and legal issues. We have digitized the contents of these more than 60 physical folders in recent months.

When the topic gained momentum in the media in the mid-1990s, it slowly but steadily became an archive of folders. Scanning has now made a huge amount of information accessible in a way that makes research and analysis possible in the first place. A total of 8,381 documents comprising 16,276 pages were saved during the digitization process.

The folders contained newspaper articles, cut-outs or entire pages, as well as articles from magazines (and entire special issues) and brochures of all kinds. Many legal texts and their drafts (flags) were available as A4 dossiers, as were studies or student papers. There were also position papers and press releases with a smaller number of pages. Finally, there were flyers, posters and many small trouvailles: from hemp bags to filters, pins, stickers and so on.

At the end of the 1980s and beginning of the 1990s, the open drug scene in Zurich was the main topic in the media. The topic of hemp - previously almost exclusively in the form of hashish - only became more prominent in the media with the hemp fields and the advertisements for them in the mid-1990s, then also as hemp and no longer (only) as hashish. Sometimes, however, there were also headlines describing “hash cultivation”. Media professionals at the time (like many at the time, including THC users) did not know the difference between hemp, hash and weed.

From 1996, a group of people regularly collected articles and so the archive grew steadily. In order not to lose the overview, Sven set up the thematic archive around the turn of the millennium: Old Hemp Archive, Politics, Legal Aid, International and Miscellaneous.

From around 2005, regular archive evenings took place, where he and Priska filed the newly arrived articles and information (especially those from Ruth).

In the past, you had to work with large special scanners if you wanted to digitize a lot of documents. Today, some of you already carry the necessary technology in your trouser pockets: the iPhone 15 can take photos with a resolution of 48 megapixels! We bought this model and will continue to use it as our club cell phone after the scanning project. In addition to the newly purchased iPhone, Michael's mini server offers hundreds of gigabytes of fast SSD storage space to safely store all the original scans. We connect the iPhone via WLAN, whereas we have laid a network cable for our Mac Mini. This allows the image files, which weigh around 15 MB, to flow quickly from the cell phone camera to the server and then to the computer for previewing and combining several individual pages into a multi-page document. Our fast Internet connection has made it possible to back up the entire archive in a decentralized manner.

We use our own software, which optimally supports our scanning process and stores the resulting files efficiently and reliably.

We used a plastic box so that the camera is always in the same position and at the same distance from the document (see picture on this page). The phone is mounted outside the box, and we mounted a 1,400 lumen lamp inside the box to illuminate the documents as well as possible.

Fabian scanning one of over 8,000 documents

Basically, it's all about the balance between quality and quantity. Fabian's motto was to work as precisely as possible, but also as quickly as possible:

“Right from the start, I was aware that this would be important. With the amount of scans, you have to scan quite quickly, otherwise it all takes far too long. However, the digital archive isn't much use to us if the quality isn't right - in other words, many scans are (too) crooked or the paper doesn't lie flat. Of course, nowadays some things can be corrected and adjusted afterwards - but I wanted to minimize the loss of time due to errors or multiple attempts.

On the second day, I was already able to simplify and speed up the process by lifting the box only once to change the items to be scanned. I scanned the folders from back to front so that they were filed chronologically from old to new. The actual workflow was as follows: Put the items in, 'fine-tune' them through the opening on the side, trigger the scan, meanwhile put the already scanned items in a box, check on the screen, complete the scan (save to the folder structure), take out the next item, take out the scanned item, put in the new one and start again.

I quickly got used to the black fingers and was surprised that my left arm or hand never hurt. After a few scans, I fell into a kind of trance, was “in the flow” so to speak and almost unstoppable. Sometimes the box had to be adjusted again, but I soon did that automatically. Folding back large newspaper articles was particularly tedious (because it was time-consuming), but the small folds went pretty quickly.

Fortunately, we were able to borrow a document scanner, because without such a device, scanning the large A4 dossiers would hardly have been possible or would have taken far too long and required too much storage space. Certain magazines and booklets had to be cut to size in advance so that they could be scanned with this device in the first place.”

We automatically crop the original scans to their content (technical term “to crop”). To do this, we use “OpenCV”, an open source solution that contains various algorithms for computer vision. What worked best was to first make the high-resolution original image less clear with “Morphological Erosion” (as it is only about the rough structure!) and then to capture the transitions with “Canny Edge Detection”:

On the left is an original scan, on the right is an intermediate result of the OpenCV algorithms.

Our program then searches for the start of the content from the top, bottom, left and right, ignoring the frame of the box. Finally, we create a new, cropped image file. A few documents have to be cropped manually, but we have cropped the vast majority of documents automatically. The difference in the amount of data is enormous: 140 GB of originals became 40 GB of cropped files.

Here's a tip for anyone who wants to implement a similar project: Use a matte background of the same color (without reflections!) to create ideal conditions for cropping. Then a color filter, as with a green screen, may be all you need and the image data is easy game for compression methods such as JPG, resulting in smaller original files.

Now that we have digitized over 16,000 individual pages, the next development is an easy-to-use interface to navigate through this amount of data. The archive is divided into five sections: Old Hemp Archive, International, Politics, Legal Aid and Miscellaneous. Our archive browser displays these five main areas as well as the respective subfolders and allows you to either browse through all documents or further restrict them as you wish. For example, you could start a search in the Politics section, then select the Hemp Initiative folder and then browse through all the documents in the Counterproposal subfolder.

Navigating in the hierarchy is useful, but a search function would of course be even more practical! The data is currently available as image files, but in the next post-processing step we want to convert this data into text form (technical term “OCR”, Optical Character Recognition), which makes a search function possible. The difficulty here is to first capture the structure of the scanned content, i.e. where is a graphic located, where do the text columns of the surrounding article begin, how do they break up and so on. Only after we have extracted the text from the structure do OCR programs deliver good results. With the text version, we can not only implement useful functions such as full-text search, but also further steps such as automatic date recognition or later summarizing and combing through the content with AI systems.

Our association archive serves our legal advice, the articles in our magazine and the memory of our development as Association.

By the way: If you have any interesting information, please send it to us! We will continue to add content to our archive in the future and develop our archive further.