Friday, March 14, 2008

A short history of file sharing in Japan

Updated to 2018

Although FTP, IRC and Usenet had come into use by the 1980's, it wasn't until the appearance of Shawn Fanning's Napster in 1999 that file sharing began to reach a mass audience. Napster was an application that users could download for free, which allowed them to browse, and download each other's mp3's. By February 2001, 26 million people were using Napster. Other applications soon followed.

In March 2000, Nullsoft, the subsidiary of AOL which makes Winamp, released the first version of a client called Gnutella, and the Dutch company Consumer Empowerment released Kazaa Media Desktop connecting to the same FastTrack network as Morpheus. After a day of intensive downloads, AOL removed Gnutella from their site, but developers soon began working on creating compatible clients that could connect to the decentralized network.

In Japan, in December 2000, the Download Soft board was created on 2 channel. This would be the place where WinMX and later Share hashes were posted along with the first Winny executable.

In early 2001, another American company, Frontcode Technologies led by Kevin Hearn released an application called WinMX. Originally, WinMX was based on the Napster protocol, so that its client could connect to independent OpenNap servers, which had started to grow up. In May, however, Frontcode added the ability to connect to a network of its own, the WinMX Peer Network. Unlike Napster, WinMX was designed to handle languages that are encoded using double bytes such as Japanese, and this led to WinMX becoming the first real mass file sharing client in Japan. One of WinMX's innovations was the ability to search by hash, a long string of numbers and letters that uniquely identifies each file on the network. This allows users to rename files as they please, while still preserving the ability to find identical files. WinMX users in Japan flocked to Japanese message boards such as 2channel to post hashes for their favorite files. From 2002 until 2006, WinMX was the most popular client for p2p in Japan.

In the meantime, several record companies had launched a lawsuit against Napster alleging that the software contributed to copyright infringement. Napster lost the case, and in July 2001, finding itself unable to create filters to block copyrighted material, Napster shut down its servers.

Later that year, Limewire LLC released its own client based on the Gnutella protocol. Like WinMX, from fairly early on, Limewire was capable of handling Japanese characters.

In Japan as well, a WinMX-using Japanese student was arrested, and had his computer confiscated, so Japanese developers began to work on the problem of developing an anonymous client. The west already had one client known for its anonymity, Freenet, and this captured the attention of one Isamu Kaneko, a grad student at Tokyo University. While Napster needed central servers to keep track of who is sharing what file, Freenet is made up of a network of co-equal "nodes" (clients which are connected to the network). When you search for a "key" (a hash value), Freenet spiders out to neighboring nodes looking for "clusters" which contain related keys. Kaneko adapted these basic ideas, and in spring of 2002, released Winny onto the download board on 2channel. Winny allows users to input three cluster keywords based on the type of file they are looking for, so they can more easily connect to people with those same interests. Winny also introduced the concept of "triggers." If you set up a trigger for a particular filename or hash, Winny will automatically try to download any file that matches. Triggers are necessary because Winny has no central search engine, but if they are too broad, you can end up downloading a lot of unwanted files just because they happen to have the same keyword in the title.

Despite the supposed anonymity of Winny, in November 2003, two Winny users were arrested by the Kyoto police. It appears that they had posted on Winny's bulletin board the names of files they were uploading, and police managed to trace their IP addresses through their posts, and they were arrested. The Winny community continued to post file hashes to 2channel, but it seems that it was this arrest which led WinMX users to stop posting hashes.

In the search for greater anonymity, in January 2004, the developer Sonchou released the first version of Share, a peer-to-peer application which tried to improve on Winny. One key feature was the ability to use DB-only triggers to capture search results from all nodes with which Share comes into contact. This helps users find the specific hash for the file they are looking for, without having to download unwanted files as in Winny.

In May 2004, Isamu Kaneko was arrested on charges of actively encouraging copyright infringement. He was released on bail a few months later, sentenced to a year in prison, and asked to pay a fine of 1.5 million yen in 2006. He is currently appealing his sentence.

In November 2004, a group of Japanese developers released Cabos, a client for the Gnutella network based on Limewire.

By 2005, WinMX had clearly established itself as the most popular file sharing client in the world with an audience of over 2 million regular users. In September, the Recording Industry Association of America issued a cease and desist letter to Frontcode ordering them to implement filters, or shut down. Frontcode responded by shutting down on September 21st. By the 25th, two separate groups of developers had released patches that changed WinMX's DNS lookup so it could find current peer caches. It seems that Japanese WinMX users flocked back to the network, while many p2p users in the west moved on in search of a new network, ultimately Bittorrent.

2007 saw Winny overtake WinMX as the most popular client in Japan. Gnutella clients had also gained a following.

In March 2007, the software company Retina released Sharebot, a program designed to trace which users have uploaded particular files on the Share network. Rumors began spreading that Share's anonymity had been compromised, but actually, Share users soon learned how to set up a plugin called DiffusionProClone to block Sharebot from scanning one's uploads. Other plugins such as sharebotblocker and connexoblocker have recently been released with the same purpose.

From 2007 to 2010, there was a Japanese language forum for torrents: torrent-jp.com/forum.

2008 seems to have been the year when p2p reached its peak in Japan. The Recording Industry Association of Japan reported seeing 209 000 people connect to Share, and 181 000 connect to Winny in that year's report.

In 2009, the Japanese language website Alabout came online, providing torrents or Share hashes for Japanese adult videos.

In 2010, the Japanese Diet outlawed downloading for the first time, and in 2012, penalties were strengthened. P2P usage on Perfect Dark and Share seems to have declined after this, with the Winny user base shrinking less so.

Before 2013, the various watchdog agencies don't seem to have paid much attention to Bittorrent, but that year the amount of bandwidth being used by Bittorrent seems to have overtaken Perfect Dark and Share.

3 comments:

Anonymous said...

Nice blog!
I hope you can post some tutorials
for Perfect Dark, with pictures
if possible.

Read you later XD

Shirley said...

I'm currently writing my MA thesis on Internet piracy in Japan and really enjoyed this article! I thought I'd point out an important network from the history of file sharing in Japan (which will remain history): File Rogue.

Loving this. I don't guess you'd be willing to share any of your sources?

Fuurin Kazan said...

I've added some links to various Japanese sources.