In CCTV context a protocol is the command set used to control one device from another. An example is that each manufacturer develops their own code to send commands to PTZ domes. This is why you see so many options for camera control in DVRs. The protocol is the software element. The hardware element is RS-422, RS- 485, etc.
A Protocol defines communication rules between devices. Surveillance systems use protocols like ONVIF, RTSP, or HTTP.