I think that the only consistent answer is that no, you CAN'T just abscond with another sites data and use it for commercial purposes... and yes, search engines are 'commercial purposes'. However, with search engines, a good case can and should be made that they are already common practice, and that it is well known how to stop polite search engines from searching your site by using a standardized file called robots.txt.