Asumming for both server and client a restrictive INPUT and open OUTPUT, i.e.:
iptables -P INPUT DROP
iptables -P OUTPUT ACCEPT
And from iptables-extensions(8) over the example of FTP in active mode:
1. NEW
NEW The packet has started a new connection or otherwise
associated
with a connection which has not seen packets in both directions.
The client on port 50000 (any random unprivileged port) connects to FTP server on port 21, the server would need at least this to accept this incoming connection:
iptables -A INPUT --dport 21 -m state --state NEW -j ACCEPT
2. ESTABLISHED
ESTABLISHED
The packet is associated with a connection which has seen
packets in both directions.
Now on the client side, he opened an outgoing connection to server on port 21 using a local port 50000 and he needs the following iptables to allow the response to arrive from server (21) to client (50000):
sudo iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
3. RELATED
RELATED
The packet is starting a new connection, but is associated with
an existing connection, such as an FTP data transfer or an ICMP
error.
Now after the FTP connection has been established and a data connection is about to be performed, client will open a server socket (yes, with active FTP client becomes a server for the data connection) on port 60000 (to my understanding client will mark this port 60000 as RELATED to the other connection from 50000->21) and will send this port number to server using the FTP PORT command. Then the FTP server will open a new connection from its port 20 to port 60000 on the client, and well, client now requires the following to allow this new connection to succeed:
sudo iptables -A INPUT -m state --state RELATED -j ACCEPT
Finally, for this to work you need to enable ip_conntrack_ftp kernel module to allow the system to mark connections/packages as RELATED (this is my understanding, I haven't digged on this too much):
modprobe ip_conntrack_ftp