Friday, March 14, 2008

A short history of file sharing in Japan

Although FTP, IRC and Usenet had come into use by the 1980's, it wasn't until the appearance of Shawn Fanning's Napster in 1999 that file sharing began to reach a mass audience. Napster was an application that users could download for free, which allowed them to browse, and download each other's mp3's. By February 2001, 26 million people were using Napster. Other applications soon followed. In March 2000, Nullsoft, the subsidiary of AOL which makes Winamp, released the first version of a client called Gnutella, and the Dutch company Consumer Empowerment released Kazaa Media Desktop connecting to the same FastTrack network as Morpheus. After a day of intensive downloads, AOL took down Gnutella, but developers soon began working on creating compatible clients for the same network.

Then in early 2001, another American company, Frontcode Technologies released an application called WinMX. Originally, WinMX was based on the Napster protocol, so that its client could connect to independent OpenNap servers, which had started to grow up. In May, however, Frontcode added the ability to connect to a network of its own, the WinMX Peer Network. Unlike Napster, WinMX was designed to handle languages that are encoded using double bytes such as Japanese, and this led to WinMX becoming the first real mass file sharing client in Japan. One of WinMX's innovations was the ability to search by hash, a long string of numbers and letters that uniquely identifies each file on the network. This allows users to rename files as they please, while still preserving the ability to find identical files. WinMX users in Japan flocked to Japanese message boards such as 2channel to post hashes for their favorite files.

In the meantime, several record companies had launched a lawsuit against Napster alleging that the software contributed to copyright infringement. Napster lost the case, and in July 2001, finding itself unable to create filters to block copyrighted material, Napster shut down its servers.

Later that year, Limewire LLC released its own client based on the Gnutella protocol. Like WinMX, from fairly early on, Limewire was capable of handling Japanese characters.

In Japan as well, a WinMX-using Japanese student was arrested, and had his computer confiscated, so Japanese developers began to work on the problem of developing an anonymous client. The west already had one client known for its anonymity, Freenet, and this captured the attention of one Isamu Kaneko, a grad student at Tokyo University. While Napster and WinMX need central servers to keep track of who is sharing what file, Freenet is made up of a network of co-equal "nodes" (clients which are connected to the network). When you search for a "key" (a hash value), Freenet spiders out to neighboring nodes looking for "clusters" which contain related keys. Kaneko adapted these basic ideas, and in spring of 2002, released Winny onto the download board on 2channel. Winny allows users to input three cluster keywords based on the type of file they are looking for, so they can more easily connect to people with those same interests. Winny also introduced the concept of "triggers." If you set up a trigger for a particular filename or hash, Winny will automatically try to download any file that matches. Triggers are necessary because Winny has no central search engine, but if they are too broad, you can end up downloading a lot of unwanted files just because they happen to have the same keyword in the title.

Despite the supposed anonymity of Winny, in November 2003, two Winny users were arrested by the Kyoto police. It appears that they had posted on Winny's bulletin board the names of files they were uploading, and police managed to trace their IP addresses through their posts, and they were arrested. The Winny community continued to post file hashes to 2channel, but it seems that it was this arrest which led WinMX users to stop posting hashes.

In the search for greater anonymity, in January 2004, the developer Sonchou released the first version of Share, a peer-to-peer application which tried to improve on Winny. One key feature was the ability to use DB-only triggers to capture search results from all nodes with which Share comes into contact. This helps users find the specific hash for the file they are looking for, without having to download unwanted files as in Winny.

In May 2004, Isamu Kaneko was arrested on charges of actively encouraging copyright infringement. He was released on bail a few months later, sentenced to a year in prison, and asked to pay a fine of 1.5 million yen in 2006. He is currently appealing his sentence.

In November 2004, a group of Japanese developers released Cabos, a client for the Gnutella network based on Limewire.

By 2005, WinMX had clearly established itself as the most popular file sharing client in the world with an audience of over 2 million regular users. In September, the Recording Industry Association of America issued a cease and desist letter to Frontcode ordering them to implement filters, or shut down. Frontcode responded by shutting down its servers on September 21st. By the 25th, two groups of developers had set up new servers, and released patches allowing users to reconnect. It seems that Japanese WinMX users flocked back to the network in droves, while most p2p users in the west moved on in search of a new client.

In March 2007, the software company Retina released Sharebot, a program designed to trace which users have uploaded particular files on the Share network. Rumors began spreading that Share's anonymity had been compromised, but actually, Share users soon learned how to set up a plugin called DiffusionProClone to block Sharebot from scanning one's uploads. Other plugins such as sharebotblocker and connexoblocker have recently been released with the same purpose.

Winny and WinMX have long been the two most popular file sharing applications in Japan, but slowly Gnutella clients such as Limewire and Cabos have been gaining more of an audience. In December 2007, the Recording Industry Association of Japan released its annual report on file sharing. 2007 saw a massive explosion in p2p use over the previous year. The most popular clients/networks were

1. Winny
2. Limewire
3. WinMX
4. Cabos
5. Share
6. BitTorrent (i.e. BitComet, Azureus and utorrent)
7. Freenet
8. Kazaa
9. Perfect Dark

Winny and Limewire are most popular among casual users who download a small number of mp3's each year. WinMX was the most popular among heavy users and those sharing adult files. BitTorrent has a smallish audience who use it to download Japanese adult videos seeded mainly by Chinese fans. Lately, Share seems to be picking up a lot of the more serious Winny users.


Anonymous said...

Nice blog!
I hope you can post some tutorials
for Perfect Dark, with pictures
if possible.

Read you later XD

Shirley said...

I'm currently writing my MA thesis on Internet piracy in Japan and really enjoyed this article! I thought I'd point out an important network from the history of file sharing in Japan (which will remain history): File Rogue.

Loving this. I don't guess you'd be willing to share any of your sources?