java爬虫使用ip代理：代码实战教你实现IP轮换采集

为什么Java爬虫需要代理IP

做网络数据采集的朋友都知道，同一个IP频繁访问目标网站很容易被限制访问。这就像一个人反复进出同一个商店，店员很快就能认出来。使用代理IP相当于每次进店都换一身装扮，这样就能避免被识别和限制。

神龙IP代理提供的服务正好能解决这个问题。他们拥有覆盖200多个城市的IP资源，总量超过1000万个，这意味着你可以获得充足的IP资源进行轮换使用。这些IP都是自营机房的纯净资源，配合先进的加密算法，既能保证采集效率，又能确保操作安全。

搭建基础的Java爬虫框架

我们先来构建一个简单的爬虫基础框架。这里使用HttpClient作为HTTP客户端，它是Java领域最常用的网络请求库之一。

import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

public class BasicCrawler {
    public static String fetchData(String url) {
        try (CloseableHttpClient httpClient = HttpClients.createDefault()) {
            HttpGet request = new HttpGet(url);
            
            try (CloseableHttpResponse response = httpClient.execute(request)) {
                HttpEntity entity = response.getEntity();
                return EntityUtils.toString(entity);
            }
        } catch (Exception e) {
            e.printStackTrace();
            return null;
        }
    }
    
    public static void main(String[] args) {
        String data = fetchData("目标网站URL");
        System.out.println(data);
    }
}

代理IP的集成与配置

现在我们来为爬虫添加代理IP功能。神龙IP代理支持多种协议，这里我们以HTTP代理为例进行演示。

import org.apache.http.HttpHost;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;

public class er {
    public static CloseableHttpClient createProxyClient(String proxyHost, int proxyPort) {
        HttpHost proxy = new HttpHost(proxyHost, proxyPort);
        
        RequestConfig config = RequestConfig.custom()
                .setProxy(proxy)
                .setConnectTimeout(5000)
                .setSocketTimeout(5000)
                .build();
                
        return HttpClients.custom()
                .setDefaultRequestConfig(config)
                .build();
    }
    
    public static String fetchWithProxy(String url, String proxyHost, int proxyPort) {
        try (CloseableHttpClient httpClient = createProxyClient(proxyHost, proxyPort)) {
            HttpGet request = new HttpGet(url);
            
            try (CloseableHttpResponse response = httpClient.execute(request)) {
                return EntityUtils.toString(response.getEntity());
            }
        } catch (Exception e) {
            e.printStackTrace();
            return null;
        }
    }
}

实现智能IP轮换策略

单一的代理IP使用一段时间后同样可能被限制，因此我们需要实现IP轮换机制。神龙IP代理的动态高级套餐非常适合这种场景，日更200万+IP资源可以满足频繁轮换的需求。

import java.util.ArrayList;
import java.util.List;
import java.util.Random;

public class IPRotator {
    private List ipPool;
    private Random random;
    
    public IPRotator() {
        this.ipPool = new ArrayList<>();
        this.random = new Random();
        // 这里应该从神龙IP代理API获取IP列表
        initializeIPPool();
    }
    
    private void initializeIPPool() {
        // 模拟从神龙IP代理服务获取IP列表
        ipPool.add("112.84.178.121:8080");
        ipPool.add("183.196.123.45:8080");
        ipPool.add("121.239.167.89:8080");
        // ... 更多IP
    }
    
    public String getRandomIP() {
        if (ipPool.isEmpty()) {
            throw new IllegalStateException("IP池为空");
        }
        return ipPool.get(random.nextInt(ipPool.size()));
    }
    
    public void rotateIP() {
        // 在实际应用中，这里应该调用神龙IP代理的API更换IP
        System.out.println("正在更换代理IP...");
    }
}

完整的代理IP爬虫实现

将前面各个模块整合，我们得到一个完整的带代理IP轮换功能的爬虫：

public class Advanceder {
    private IPRotator ipRotator;
    private int requestCount = 0;
    private static final int MAX_REQUESTS_PER_IP = 50;
    
    public Advanceder() {
        this.ipRotator = new IPRotator();
    }
    
    public String smartFetch(String url) {
        if (requestCount >= MAX_REQUESTS_PER_IP) {
            ipRotator.rotateIP();
            requestCount = 0;
        }
        
        String currentIP = ipRotator.getRandomIP();
        String[] ipParts = currentIP.split(":");
        String proxyHost = ipParts[0];
        int proxyPort = Integer.parseInt(ipParts[1]);
        
        String result = er.fetchWithProxy(url, proxyHost, proxyPort);
        requestCount++;
        
        return result;
    }
    
    public static void main(String[] args) throws InterruptedException {
        Advanceder crawler = new Advanceder();
        
        for (int i = 0; i < 200; i++) {
            String data = crawler.smartFetch("目标网站URL");
            System.out.println("第" + (i+1) + "次请求结果长度: " + 
                (data != null ? data.length() : "请求失败"));
            
            // 添加随机延迟，模拟更真实的访问行为
            Thread.sleep(1000 + new Random().nextInt(2000));
        }
    }
}

错误处理与性能优化

在实际使用中，我们需要考虑各种异常情况并优化性能：

public class Robuster {
    public static String robustFetch(String url, String proxyHost, int proxyPort, int retryCount) {
        for (int i = 0; i < retryCount; i++) {
            try {
                String result = er.fetchWithProxy(url, proxyHost, proxyPort);
                if (result != null && !result.trim().isEmpty()) {
                    return result;
                }
            } catch (Exception e) {
                System.out.println("第" + (i+1) + "次尝试失败: " + e.getMessage());
                
                if (i < retryCount - 1) {
                    try {
                        Thread.sleep(2000  (i + 1)); // 指数退避
                    } catch (InterruptedException ie) {
                        Thread.currentThread().interrupt();
                    }
                }
            }
        }
        return null;
    }
}